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1.0  INTRODUCTION 

Augmented  Reality  (AR)  and  Virtual  Environments  (VE)  have  been  introduced  in  the  late  80s  as 
innovative  systems  for  displaying  synthetic,  computer-generated  environments  as  if  they  were  real. 
Various  possible  applications  were  identified  soon,  for  instance:  Product  design,  architecture,  training,  or 
teleoperation.  But  it  turned  out  that  computers’  rendering  power  and  display  technology  were  too  limited 
for  a  reasonable  application  then.  The  whole  concept  seemed  to  be  a  promising  science-fiction  topic  that 
had  come  up  too  early  for  realization  and  practical  use. 

Nonetheless,  technology  evolved  much  faster.  Less  than  a  decade  later,  systems  were  powerful  enough  for 
first  sample  applications.  Due  to  the  huge  demands  on  computing  and  rendering  power  (and  resulting  high 
financial  expenses)  they  were  just  a  small  number  of  institutions  doing  research  in  this  area.  But  again,  it 
took  only  a  short  time  until  low-cost  alternatives  became  available.  The  user  community  became  much 
broader  and,  consequently,  research  and  application  activities  were  spread  and  intensified. 

But  while  computing,  rendering,  and  display  technologies  have  made  tremendous  advance  the  design  of 
the  human-system-interface  (HSI)  did  not.  As  a  matter  of  fact,  most  systems  still  have  to  be  run  by  special 
trained,  qualified  personnel,  and  most  applications  are  often  limited  to  passive  presentations.  The  systems’ 
interfaces  are  often  prototypic  and  adapted  from  conventional  two-dimensional  graphical  human- 
computer-interfaces.  Interfaces  remain  limited  to  an  extension  of  existing  concepts  without  exploiting  their 
real  potential.  A  better  integration  of  new  concepts  for  training,  for  system  design  and  for  other  future 
applications  by  including  the  capabilities  of  systems  in  an  early  conceptual  stage  would  be  desirable.  For 
this,  the  HSI  has  to  be  ergonomically  designed  on  different  levels,  including  the  pragmatic,  semantic, 
syntactic,  and  even  lexical  level. 

It  turns  out  that  the  ergonomic  design  of  HSI  is  an  intensifying  critical  issue.  The  former  NATO  Research 
Task  Group  on  Human  factors  in  virtual  environments  (HFM-021)  has  identified  Virtual  Reality  (VR) 
or  Virtual  Environments  (VE)  to  be  advantageous  for  a  close,  natural,  and  widely  intuitive  interaction  by 
making  better  use  of  the  human  perceptive,  cognitive,  and  motor  capabilities.  It  defined  VR  as  the 
experience  of  being  in  a  synthetic  environment  and  the  perceiving  and  interacting  through  sensors  and 
effectors,  actively  and  passively,  with  it  and  the  objects  in  it,  as  they  were  real.  VR  technology >  allows  the 
user  to  perceive  and  experience  sensory’  contact  and  interact  dynamically  with  such  contact  in  any  or  all 
modalities. 

During  its  three  years  work  it  found  that  VE  have  developed  into  a  useful  technology  for  various 
applications.  Most  of  them  were  found  in  product  design,  product  presentation,  education,  training,  and 
visualization  of  massive  datasets.  First  experimental  approaches  of  integrating  VE  in  military  education 
and  training  as  well  as  mission  rehearsal,  support,  and  analysis  were  also  identified  (NATO  RTA,  2001). 

Potential  future  military  applications  of  VE-systems  identified  in  the  areas  of  (NATO  RTA,  2003): 

•  Human-system  interfaces  in  materiel  design  (virtual  prototyping). 

•  Command,  control,  communications,  computer,  intelligence,  surveillance,  reconnaissance 
(C4ISR). 

•  Telepresence,  teleoperation  and  telemanipulation  in  reconnaissance,  surveillance  and  target 
acquisition. 
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•  Realistic  and  distributed  simulation  and  training. 

•  Short-term  mission  preparation,  including  intelligent  synthetic  individuals  and  intended  area  of 
operation. 

•  Mission  support  as  wearable,  augmenting  technology  (including  Mixed  and  Augmented  Reality). 

In  2003  the  NATO  Research  Task  Group  on  Advances  of  Virtual  Environments  for  Human-System- 
Interaction  (HFM-121)  was  established  in  order  to  update  the  actual  state-of-the-art  of  Augmented, 
Mixed,  and  Virtual  Environments  (AMVE)  and  derive  recommendations  for  the  further  development  and 
application  of  this  technology.  Within  this  group,  experts  from  Canada,  Denmark,  France,  Germany,  the 
Netherlands,  Sweden,  United  Kingdom,  and  USA  are  exploring  the  potential  of  VE  systems  as  an 
innovative  technology  for  effective  and  efficient  human-system  interaction  and  to  investigate  system 
capabilities  in  the  military  domain. 

First,  a  report  on  the  actual  state  of  research  and  technology  within  participating  nations  was  compiled. 
The  present  technical  report  subsumes  the  German  part  of  this  compendium  by  giving  an  overview  over 
research  institutions  with  a  primarily  military  focus,  larger  defensive  projects  including  AMVE 
technology,  and  over  civil  applications  of  AMVE  technology. 

Generally  said,  in  Germany  AMVE  research  is  spread  across  a  diversity  of  civil  and  military  agencies  and 
institutions.  Most  military  projects  concerning  AMVE  topics  are  coordinated  by  the  German  Ministry  of 
Defence  and  subordinate  agencies.  They  are  submitted  to  research  institutions,  academic  institutes,  and 
defence  companies. 

In  this  area,  AMVE  is  not  considered  to  be  a  high-level  topic  on  its  own.  Instead,  it  always  refers  to 
special  applications  and  larger  defence  program.  A  key  domain  is  traditionally  simulation  in  military 
education  and  training.  Another  domain  is  design  of  military  materiel,  especially  in  connection  with 
visualization,  ergonomic  analyses  and  computer-aided  design.  Innovative  approaches  of  applying  AMVE 
technologies  as  an  innovative  HSI  and  for  teleoperation  also  came  up  recently.  These  domains  widely 
interfere  with  academic  and  civil  research  so  that  there  is  a  close  connection  and  exchange  of  knowledge 
with  academic  and  civil  research  projects.  Dual-use  has  become  a  common  term  for  characterizing  this 
sharing  of  results  across  domains. 

This  compendium  summarizes  current  military  projects  including  AMVE  topics  in  Germany.  Other 
activities  in  military  simulation,  for  instance  flight  or  tank  simulators,  were  intentionally  left  out  because 
they  were  not  within  the  focus  of  the  NATO  RTG.  Hence,  a  technology-based  approach  of  differentiating 
between  AMVE  and  traditional  simulation  will  be  given  in  chapter  7.1.  This  report  includes  three  parts. 
First,  studies  and  activities  of  institutional  research  and  within  defence  industry  are  presented.  Chapters  2, 
3,  and  4  handle  this  part.  Second,  main  military  projects  are  described  briefly  in  chapters  5  and  6.  Finally, 
in  chapter  7,  the  results  of  a  questionnaire  survey  on  civil  AMVE  activities  in  Germany  are  depicted. 

1.1  References 

NATO  RTA  (2001):  What  Is  Essential  for  Virtual  Reality  Systems  to  Meet  Military:  Human  Performance 
Goals?  Meeting  Proceedings  RTO-MP-058.  Neuilly-sur-Seine:  NATO  RTA. 

NATO  RTA  (2003):  Virtual  Reality:  State  of  Military  Research  and  Applications  in  Member  Countries. 
Technical  Report  RTO-TR-018.  Neuilly-sur-Seine:  NATO  RTA. 
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2.0  RESEARCH  INSTITUTE  FOR  COMMUNICATION,  INFORMATION 
PROCESSING,  AND  ERGONOMICS  (FGAN  -  FKIE) 

T.  Alexander 

2.1  Areas  of  Interest 

•  Providing  scientific  and  technical  expertise  for  the  German  Ministry  of  Defence  and  subordinate 
agencies  in  the  area  of  C4ISR  and  innovative  HSI. 

•  Applied  research  in  the  area  of  human  factors  in  military  AMVE-Systems. 

2.2  Current  Research  Projects 

2.2.1  Applicability  of  AMVE  Technology  as  Tactical  Situation  Display  (TSD) 

The  general  goal  of  this  project  is  to  develop  recommendations  for  the  applicability  and  usability  of 
stationary  and  wearable  AMVE-technology  as  advanced  tactical  situation  display  of  C4ISR  systems. 
The  results  are  included  into  the  development  of  different  prototypes  for  such  display  systems.  Examples 
of  these  are  shown  in  Figure  2-1. 


Figure  2-1:  Powerwall  TSD,  Electronic  Sandtable,  Portable  TSD  for  Dismounted  Soldiers. 


The  designs  take  into  account  changes  of  demands  on  today’s  C4ISR  systems.  Within  the  scope  of  future 
scenarios  there  will  be  a  high  demand  on  detailed  and  actual  information  in  C4ISR.  It  is  often  (and  falsely) 
inferred  that  more  information  always  leads  to  better  tactical  decision-making  and  higher  mission  success. 
High-tech  data  acquisition,  transfer,  and  pre-processing  will  enable  the  military  commander  to  collect  a 
huge  variety  of  diverse  information  about  the  current  situation.  But  this  massive  quantity  of  information  is 
also  risky;  especially  in  time-critical  situations  with  tactical  decision-making  under  stress  relevant 
information  is  likely  to  be  overseen.  This  leads  to  wrong  decisions  with  far-reaching,  sometimes  fatal 
consequences.  It  can  be  avoided  by  making  better  use  of  the  human  perceptive,  cognitive,  and  motor 
capabilities.  AMVE  technology  can  be  applied  here  and  is  very  promising  already. 

The  human  factors  have  to  be  considered  as  early  as  possible  when  designing  and  evaluating  new  display 
systems.  Thus,  detailed  research  on  the  display  system  itself,  e.g.  on  characteristics  of  the  display 
(stereoscopy,  head-tracking,  resolution,  ocular  stress)  and  interaction  type  (relative/absolute  interaction, 
device,  input  signal  gain)  and  their  effects  on  human  performance,  as  well  as  research  on  the  kind  of 
information  visualization  itself  (possibilities  for  visualizing  time  tracks  and  uncertainty)  are  carried  out 
within  diverse  projects  in  this  field. 

The  results  are  integrated  into  prototypic  designs  of  AMVE  tactical  situation  displays  for  the  tactical 
decision-making  on  different  military  commando  levels,  from  operational  to  tactical  command  hierarchy. 


RTO-TR-HFM-1 21 -Part-1 


3-3 


VIRTUAL,  MIXED,  AND  AUGMENTED  SURVEY  -  GERMANY 


ORGANIZATION 


2.2.2  Visualization  of  Massive  Amounts  of  Abstract  Data 

The  technologic  and  thematic  development  in  the  area  of  surveillance  and  intelligence  is  leading  to 
growing  amounts  of  inhomogeneous  information  and  to  changing  analysis  procedures.  While  in  the  past 
analogue  radio  networks  allowed  a  clear  distinction  and  localization  of  transmitting  and  receiving  unit  this 
is  not  possible  with  upcoming  worldwide,  distributed,  digital  communication  networks.  Especially  with 
growing  demands  coming  along  with  asymmetric  warfare  it  is  more  important  to  sort  out  relevant 
information  out  of  tree-speech  text  messages,  detect  correlations  to  other  messages,  and  to  visualize  the 
results  of  these  pre-analysis  in  order  to  support  the  intelligence  analyst. 

Because  of  the  huge  amount  of  messages  and  the  natural  way  of  displaying  information,  it  was  decided  to 
use  AMVE  technology  as  display.  By  integrating  models  of  the  human  spatial  processing  memory  general 
guidelines  for  visualizing  inhomogeneous  abstract  information  are  being  derived  and  implemented  into  a 
prototype  display  system.  As  shown  in  Figure  2-2,  this  system  is  scaleable  from  desktop-VR  for  today’s 
use  to  a  more  sophisticated  cooperative  VE-system  for  future  applications. 


Figure  2-2:  Information  Visualization  on  Desktop  Display  (left) 
and  Within  an  Immersive  VE  (right). 


2.2.3  Infantryman  of  the  Future 

The  Infantryman  of  the  Future  (Infanterist  der  Zukunft  -  IdZ)  is  a  large  project  succeeding  the  Soldier 
2000  and  Soldier  2010  projects  of  the  German  Ministry  of  Defence.  In  a  long-term  approach,  IdZ  includes 
augmented  reality  technology  to  support  the  dismounted  soldier  with  actual  tactical  information  in  his 
mission.  Synchronously,  the  soldier  collects  data  and  enters  it  into  a  C4ISR  system. 

In  cooperation  with  the  central  institute  for  medical  service  of  the  German  armed  forces  (ZInstSanBw) 
a  realistic  scenario  for  determining  performance  tests  and  trials  is  to  be  built  up.  In  this  connection, 
a  VE-system  is  used  to  create  realistic  scenarios  and  evoke  physiological  and  mental  workload. 

2.2.4  Teleoperation  of  Unmanned  Semi-Autonomous  Vehicles 

Remotely  piloted  vehicles  (RPV:  UGV,  UAV,  UCAV,  UUV)  are  of  growing  importance  for  present  and 
future  military  scenarios.  The  supervisory  control  of  RPVs  represents  a  cognitively  highly  demanding  task 
for  a  human  operator.  To  maintain  required  levels  of  situation  awareness  (SA)  an  operator  must  process 
and  respond  to  multiple  complex  information  streams  in  real-time.  Therefore,  the  human-RPV-interface 
plays  a  key  role  in  enabling  an  operator  to  effectively  control  an  RPV  with  respect  to  mission 
requirements.  AMVE-based  user  interfaces  have  the  potential  to  overcome  limitations  of  presently  used 


3-4 


RTO-TR-HFM-1 21  -Part-1 


VIRTUAL,  MIXED,  AND  AUGMENTED  SURVEY  -  GERMANY 


2D-interfaces.  Significant  amounts  of  information  provided  by  RPVs,  e.g.  sensor  data,  includes 
georeferenced  data,  and  can  therefore  take  advantage  from  AMVE  systems. 

2.3  Results  Achieved  to  Date 

VE  activities  at  FKIE/EFS  have  originally  started  with  usability  studies  on  VE  for  ergonomic  workplace 
design  and  analysis.  They  were  found  to  be  advantageous  for  rapid  prototyping  applications.  At  a  very 
early  design  phase,  CAD-models  of  products  could  be  visualized  and  examined  as  if  they  were  real. 
Improvements  were  easily  possible  and  the  effects  could  be  visualized  immediately. 

With  respect  to  future  tactical  situation  display  a  first  prototype  for  applying  VE  technology, 
the  Electronic  Sandtable  (ElSa)  testbed,  was  designed  and  developed.  Its  concept  is  based  on  the  sandtable 
metaphor,  which  models  the  working  procedures  of  a  traditional  sandtable  as  used  frequently  in  military 
tactical  education.  Geographic  data  and  dynamic  tactical  situation  data  are  stereoscopically  displayed  on  a 
horizontal  surface  creating  the  illusion  of  a  three-dimensional  landscape.  A  self-developed  software 
framework  handles  rendering.  It  facilitates  rendering  of  common  web3d-scenegraphs,  geo-referenced 
C4ISR-data,  and  intelligence  information  on  a  heterogeneous  computer  clusters.  Several  experiments  on 
the  sandtable  display  system  were  carried  out  to  optimize  parameters  of  the  viewing  model  and  for 
interaction. 

Because  the  original  ElSa  testbed  was  based  on  a  rather  (expensive,)  bulky  display  technology,  alternative 
displays  with  higher  availability  and  mobility  were  built  up  recently.  A  low-cost  approach  uses  the  same 
software  but  a  vertical  projection  surface  with  standard  PC  mouse  and  speech  input  for  interaction. 
The  setup  is  far  more  mobile  and  can  be  used  for  different  types  of  command  posts.  It  is  based  on  today’s 
IT-technology  and  enhances  standard  tactical  planning  procedures  -  especially  tactical  situation 
presentations. 

With  the  third  system,  which  uses  Mixed  Reality  procedures  and  is  called  MR-ElSa,  mobility  is  even 
stronger  enhanced.  Wearable  IT-technology  and  head-mounted  displays  offer  a  system’s  applicability 
“in  the  field”.  The  system  is  to  be  used  by  small  operational  units  just  before  the  mission  starts.  Common 
software  packages  were  included  into  the  VE-framework  and  a  prototype  system  has  been  developed  und 
built  up.  Interaction  procedures  are  currently  being  implemented  for  further  research  on  HIS  with  this 
display. 

2.4  VR  R&D  Laboratory 

•  Graphics  /  Rendering 

•  SGI  Onyx  2  InfiniteReality  2  Rack 

•  Inhomogeneous  COTS  PC  cluster,  diverse  NVIDIA  Geforce  graphics  cards  (permanent  updates) 

•  Xybemaut  wearable  computer 

•  Visualization 

•  Virtual  Workbench  (stereoscopic  horizontal  projection)  with  CRT-projectors 

•  Powerwall  (stereoscopic  vertical  projection)  with  LCD-projectors 

•  Diverse  Daeyang  Cy-  Visor  low-cost  stereoscopic  head-mounted  displays  (HMD)  with  self- 
developed  “video  see-through”  option 

•  Stereoscopic  Trivisio  AR  Vision3D  HMD 

•  MicroOptical  AV-1 

•  NVisor  SX 
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•  Input  Devices 

•  Electromagnetic  Ascension  Flock-of-Birds  Tracking  System 

•  Diverse  Interaction  devices 

•  Optical  ARTrack  System 

•  CyberGlove 

•  Wacom  Input  Tablets 

•  VOCON  speech  recognition 

•  Space  Mouse 


3.0  EADS  DEUTSCHLAND  GmbH,  MILITARY  AIRCRAFT 

L.  Vogelmeier 

3.1  Areas  of  Interest 

•  Enhancement  and  customization  of  Virtual  Environments  (VE)  for  use  in  System  Development  with 
focus  on  Human-Machine-Interface  (HMI)  development  and  maintainability  evaluations. 

•  Building  up  VE-based  training  devices  with  focus  on  procedure  training  for  airborne  systems  and 
virtual  flight  training. 

•  Defining  fields  of  applications  for  the  operational  use  of  VE  in  airborne  systems  and  for  airborne 
systems. 

3.2  Current  Research  Project 

3.2.1  Virtual  Reality  for  Development,  Manufacturing,  Maintenance  and  Training 

This  project  is  a  shared  research  project  inside  EADS  N.V  with  EADS  Deutschland  GmbH,  Military 
Aircraft  in  lead. 

The  main  objective  of  this  project  is  to  enhance  the  VE-Technology  to  fit  the  special  requirements  for 
development,  manufacturing,  maintenance  and  training. 

The  project  is  divided  into  three  main  work-packages: 

1)  Realistic  and  Real  Time  Capable  Light  Simulation 

The  illumination  of  cockpit  or  passenger  area  of  an  airplane  is  of  high  importance  for  low  workload  of  the 
crew  and  convenient  travelling  of  the  passengers.  Many  constraints  for  the  illumination  are  pre-defmed  in 
early  stages  of  development,  for  example  by  the  basic  geometry  of  the  aircraft  fuselage  or  the  position  of 
the  windows  and  illuminating  elements.  Therefore,  an  early  consideration  of  the  illumination  of  the 
product  is  necessary  for  an  efficient  development  process.  To  achieve  and  to  keep  development  iteration 
cycles  short,  the  use  of  computer  simulation  of  the  illumination  is  mandatory.  Techniques  must  be  found 
and  implemented  that  provide  a  computer  based  light  simulation  that  is  comparable  to  the  real  world  and 
enable  interaction  in  real  time. 

2)  Integration  of  Real  and  Virtual  Humans  into  VE 

It  is  essential  to  use  computer-simulated  data  in  a  most  efficient  way  to  decrease  time  and  costs  for 
product  development.  For  this  reason  it  is  necessary  to  find  ways  to  merge  human  characteristics 
(of  developer,  operator,  customer  of  the  product)  and  computer  data.  The  main  aspect  is  to  learn  and 
understand  how  the  human  will  act  and  sentence  the  future  product.  Two  different  concepts  are  required 
for  this.  The  first  is  to  simulate  the  person  within  the  VE  as  an  avatar,  and  the  second  is  to  integrate  a  real 
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person  into  the  VE.  The  use  of  avatars  facilitates,  for  instance,  the  opportunity  to  generate  test  persons  that 
are  not  available  for  ergonomic  studies  in  reality,  to  simulate  crowds,  or  to  make  assessments  in  dangerous 
enviromnents  without  endangering  real  persons.  Real  persons  are  still  required  to  give  feedback  about 
sensations  like  comfort  or  discomfort,  to  investigate  stress,  or  to  validate  the  data  generated  by 
assessments  with  the  avatars. 

3)  Integration  of  Complex  Simulations  in  VE 

The  objective  of  this  work  package  is  to  control  VE  by  complex  simulations.  The  target  applications  are 
the  simulation  of  a  full  functional  aircraft  cockpit  for  pilot  training  or  for  testing  the  Human-Machine- 
Interface  (HMI)  concept  of  the  cockpit,  to  visualize  the  behaviour  of  crowds  under  special  circumstances, 
e.g.  the  evacuation  of  an  airplane  after  an  emergency  landing  or  the  simulation  of  soldiers  under  combat 
situations.  Therefore,  large  amounts  of  information  must  be  exchanged  between  the  VE-system,  which 
provides  visualization  and  interaction,  and  the  simulation  software,  which  generates  and  handles  the 
underlying  data. 

3.3  Results  Achieved  to  Date 

•  Successful  integration  of  Virtual  Reality  in  the  cockpit  development  process  for  assessments  of  static 
cockpit  geometry  and  layout,  based  on  a  mixed  mockup  system.  First  adoption  of  developed 
techniques  in  the  Tornado  program  in  2002. 


Figure  3-1:  Virtual  Cockpit. 


•  A  first  demonstrator  model  of  a  mixed  mockup  based  virtual  flight  simulator  (with  restricted 
functionality)  was  implemented  in  2000  and  presented  at  several  air  shows.  First  prototype  of 
complete  cockpit  simulation  in  VR  was  accomplished  in  2003. 


Figure  3-2:  Mixed  Mockup  Flight  Simulator. 
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•  Evaluations  of  VE-based  maintainability  assessments  with  virtual  prototypes  have  been  performed. 
First  adoption  of  defined  methods  will  start  in  2004. 


Figure  3-3:  VR  Maintainability  Assessment. 


3.4  VR  R&D  Laboratory 

•  Graphics  /  Rendering 

•  SGI  Onyx  2  InfiniteReality  2  Rack 

•  Inhomogeneous  COTS  PC  cluster,  diverse  NVIDIA  and  ATI  graphics  cards 

•  Visualization 

•  Vertical  Back  Projection  (active  stereo,  CRT-projector,  One-Side-CAVE-Mode) 

•  HMD  Kaiser  XL 50 

•  HMD  NVISION  Datavisor  HiRes 

•  4-Side  Cave  (start  of  operation  planned  mid  of  2004) 

•  Input  Devices 

•  Electromagnetic  Polhemus  Fastrack  System 

•  Ultrasonic  Intersense  IS-900 

•  Optical  ARTrack  System 

•  CyberGlove 

•  Flying  Mouse 

•  DaimlerChysler  speech  recognition 

•  Space  Mouse 


4.0  IABG  mbH  (DEPT.  “NETWORKED  SIMULATIONS”  AT  MEPPEN) 


F.  Reiners 


4.1  Areas  of  Interest 

•  Interoperability  of  simulation  systems  (interoperability  between  simulation  systems  alone  and  between 
simulation  systems  and  other  systems,  e.g.  C2-systems). 

•  Application  of  VE  techniques  (mainly  improvement  of  user  interfaces,  e.g.  for  simulation  systems). 
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4.2  Current  Research  Projects 

4.2.1  Ergonomic  Investigations  with  VR  Techniques 

The  aim  of  this  project  is  to  find  out  if  (and  how)  it  is  possible  to  use  CAD  data  of  combat  vehicles  of  the 
German  army  for  visualization  purposes  VE.  This  kind  of  visualization  is  considered  to  be  very  useful  for 
the  following  reasons: 

1)  Ergonomic  investigations  (e.g.  of  the  interior  of  a  vehicle)  can  be  done  already  in  very  early 
design  phases,  well  before  real  prototypes  are  built. 

2)  The  areas  of  free  sight  for  the  driver  of  the  vehicle  or  for  other  persons,  e.g.  looking  out  of  the 
hatches,  can  be  examined,  without  the  need  for  cumbersome  and  expensive  measurements  using 
the  real  vehicle. 

3)  The  derivation  of  requirements  for  new  military  equipment  can  be  supported  enormously  by 
coupling  VE  systems  to  simulations  of  the  appropriate  environment.  For  combat  vehicles,  this 
environment  can  be  represented  by  combat  simulation  systems. 

Usually,  the  direct  use  of  CAD  data  for  VE  purposes  fails,  because  the  number  of  polygons  that  are 
needed  for  the  proper  description  of  the  geometric  extension  of  a  vehicle  is  very  huge,  and  an  acceptable 
visualization  e.g.  in  an  HMD  is  not  possible  anymore.  Instead,  a  significant  reduction  of  the  number  of 
polygons  is  required. 

The  available  CAD  systems  do  not  allow  the  automatic  conversion  of  CAD  data  into  formats  that  are  used 
in  VE  systems.  Therefore,  in  this  project  several  steps  were  examined  that  are  useful  in  performing  this 
conversion.  Figure  4-1  shows  different  sequences  of  steps  that  can  be  followed  to  reach  this  conversion. 


structures, 

colors, 

materials 


adjusting  polygon 
accuracy  in  CAD- 
systems 


investigations 


High  Level 


Format 


IGES,  STEP 


DXF, 

VRML,  STL... 


Figure  4-1:  Possible  Sequences  for  Converting  CAD  Data  into  VE  Formats. 


First  ergonomic  examinations  of  areas  of  free  sight  within  the  armored  transport  vehicle  BOXER  were 
done  by  using  an  HMD  connected  to  either  a  mechanical  or  magnetic  tracking  system.  Figure  4-2  shows  a 
simulated  view  into  the  interior  of  a  BOXER.  The  VE  data  have  been  derived  completely  from  the  CAD 
data  as  provided  by  the  manufacturer  of  the  vehicle. 
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Figure  4-2:  Simulated  View  into  the  Interior  of  a  BOXER  with  VE  Data  Derived  from  CAD  Data. 


4.2.2  Coupling  of  VE-Systems  with  Combat  Simulation  Systems 

As  already  mentioned  above,  it  makes  sense  to  join  VE-systems  with  combat  simulation  systems.  It  was 
decided  to  perform  this  coupling  in  a  two  step  procedure: 

1)  Coupling  of  an  HMD  to  a  driving  simulator; 

2)  Coupling  of  the  driving  simulator  to  a  combat  simulation  system. 

In  this  study,  available  systems  were  used:  An  experimental,  component-based  driving  simulator  and  the 
combat  simulation  system  PABST  2000  by  IABG. 

An  HMD  can  be  seen  as  a  special  graphical  monitor  where  the  projection  is  dependant  upon  the  position 
of  the  eye  of  the  user  captured  by  a  tracking  system.  In  the  experimental  driving  simulator,  it  can  be 
modified  by  keyboard  inputs.  The  main  focus  of  this  experiment  was  to  calculate  the  correct  view  shown 
in  the  HMD.  This  view  depends  on  the  right  supeiposition  of: 

1)  The  position  and  orientation  of  the  HMD  (derived  from  the  tracking  system); 

2)  The  position  of  the  eye  point,  relative  to  the  vehicle  (derived  from  parameters  of  the  driving 
simulator);  and 

3)  The  position  of  the  vehicle  within  the  synthetic  terrain  (derived  from  the  combat  simulation 
system). 

Figure  4-3  shows  an  example  of  a  picture  as  seen  by  the  HMD,  compared  to  the  corresponding  map  of  the 
combat  simulation  system. 
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Figure  4-3:  View  into  a  Virtual  Environment.  The  corresponding  visualization 
of  the  combat  simulation  system  PABST  2000  is  shown  in  the  insert. 

4.3  VR  R&D  Laboratory 

•  Graphics  /  Rendering 

•  PC,  NVIDIA  Geforce  graphics  cards 

•  Visualization 

•  Stereoscopic  HMD  NVISOR  SX 

•  Monitor 

•  Input  Devices 

•  Electromagnetic  Ascension  Flock-of-Birds  Tracking  System 

•  Mechanical  Shooting  Star  Technology  ADL-1 

•  Joystick 

5.0  CoCBT  (KoCUA)  -  CO-OPERATIVE  COMPUTER-BASED  TRAINING  FOR 
THE  AMPHIBIAN  M3 

5.1  Background 

CoCBT  is  a  training  device  for  the  German  Amphibian  vehicle  M3.  The  operational  spectrum  of  this 
trainer  covers  driving  practice  on  land  and  water  including  coupling  manoeuvres,  building  bridge  stumps 
and  ferrying  operations  with  large  ferries.  The  instructor  can  control  the  exercise  conditions  by  changing 
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the  variable  parameters  (e.g.  wind,  weather,  water  velocity,  technical,  and  tactical  incidents).  Real  crossing 
sites  on  the  rivers  Main  and  Weser  have  been  used  as  a  basis  for  the  virtual  exercise  areas.  A  wide  variety 
of  wheeled  and  tracked  vehicles  can  be  used  by  the  crossing  troops. 

The  operation  of  the  M3  amphibian  requires  the  co-operation  of  several  crews,  for  example  when  coupling 
amphibians  to  build  a  ferry.  The  previously  available  training  tools,  which  are  CBT  program  and  digital 
teaching  aid,  were  primarily  designed  for  single-place  training.  However,  a  comprehensive  amphibian 
training  requires  a  team-oriented  approach.  CoCBT  focuses  on  this  approach.  Figure  5-1  shows  the  setup 
of  CoCBT. 


Figure  5-1:  CoCBT  Amphibian  M3  Training  Facility. 


5.2  Technology 

The  CoCBT  Amphibian  M3  training  program  facilitates  single  individual  and  team  training.  In  a  VE, 
which  is  implemented  in  a  PC  network,  it  is  possible  to  train  a  team  of  up  to  four  students  in  the  operation 
of  the  amphibious  vehicle  M3.  Consequently,  CoCBT  covers  the  complete  spectrum  of  the  amphibious 
training.  The  crew  is  able  to  exercise  throughout  the  whole  year,  under  controlled  conditions,  and 
independently  from  the  otherwise  mandatory  availability  of  staff  and  equipment. 

The  whole  virtual  training  situation  is  presented  on  three  monitors  with  a  120°  panoramic  view. 
An  additional,  lower  monitor  is  used  to  show  the  necessary  handling  functions  (e.g.  control  stand). 
The  crewmembers  control  the  action  by  a  PC-mouse,  keyboard  and  the  original  master  pilot  (which  is  used 
for  steering  on  water).  The  ferry  commander  gives  hand  signals,  which  are  depicted  by  the  system  with  the 
use  of  data  gloves  and  a  tracking  system.  An  animated  character  presents  the  hand  signals  synchronously 
in  the  virtual  world. 

The  CoCBT  Training  System  consists  of  workstations  with  several  PCs,  which  are  linked  to  each  other 
in  a  network.  The  educational  network  helps  the  instructor  supervise  the  progress  of  the  exercise. 
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For  exercise  assessment  a  recording  system  is  available,  which  can  be  used  for  observing  the  training  at 
any  time.  The  recording  also  supports  the  instructor’s  follow-up  of  the  exercise.  The  complete  training 
system  is  based  on  a  scalable  technology,  which  allows  for  example,  to  add  more  workstations  or  to 
expand  the  panoramic  view  of  the  virtual  world  to  360°.  This  training  platform  can  also  be  used  in  the 
future  for  other  systems. 

5.3  Delivery 

The  CoCBT  training  system  was  officially  presented  to  the  Armed  Forces  Engineers  in  Minden  in 
December  2002. 

6.0  ADVANCED  AIR  DEFENCE  TRAINING  SIMULATION  SYSTEM  (AADTSS) 
-  VR  STINGER  TEAM  TRAINER 

6.1  Background 

In  the  past,  team  training  for  the  STINGER  weapon  system  was  done  in  stationary,  large-scale  dome 
trainers.  Changing  requirements,  especially  growing  demands  on  mobility  and  transportability,  made  a 
modernization  of  the  trainers  and  the  application  of  new  technology  necessary.  For  the  layout  of  the  new 
trainers  immersive  VE-technology  was  utilized  to  allow  a  commander  and  a  gunner  cooperative  training 
of  STINGER  handling.  The  system  had  to  cover  350°  horizontal  and  130°  vertical  field  of  view,  visualize 
multiple  targets  and  effects  and  facilitate  long-range  aircraft  detection  and  identification.  In  Figure  6-1  the 
system  is  illustrated. 


Figure  6-1:  VR  STINGER  Team  Trainer. 


6.2  Technology 

The  AADTSS  simulator  is  integrated  in  an  extensible  container.  Both,  commander  and  gunner  are 
equipped  with  Head-Mounted-Displays  (HMDs)  for  visual  display  of  the  VE.  For  long-range  aircraft 
detection/identification  a  high  resolution  was  required  and  the  HMD  is  capable  of  a  maximum  resolution 
of  1.280  x  1.024  pixels  per  eye.  Because  of  better  contrast  and  separation  of  surrounding  light,  the  HMDs 
are  without  see-through  option. 

The  team  members  have  to  co-operate  with  each  other  acoustically  and  optically,  but  due  to  the  HMD  they 
cannot  see  each  other.  By  modeling  both  as  avatars  this  problem  was  solved. 
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The  commander’s  actions  are  tracked  by  an  inertial  tracking  system,  whereas  the  gunner  is  tracked  by  an 
optical  tracking  system.  For  safety  reasons,  both  team  members  are  encircled  by  handrails.  This  solution 
was  necessary  because  of  the  FlMDs  without  see-trough-option. 

A  further  aspect  was  the  database  for  training.  The  generation  of  databases  is  based  on  stereoscopic 
photos.  It  allows  the  generation  of  scenarios,  targets  and  flight  paths  and  is  independent  of  the  simulator. 
The  system  consists  of  one  workstation,  the  stereo-camera-system  and  one  control-PC. 

6.3  Delivery 

After  an  initial  successful  troop  trial  phase  in  1997,  a  first  system  was  delivered  in  2000.  A  second  system 
has  been  ordered  from  the  German  Air  Force  recently.  Whereas  the  first  system  has  been  integrated  into  a 
mobile  container,  the  second  will  be  modified  for  inside  building  uses. 


7.0  SUMMARY  OF  CIVIL  AMVE  RESEARCH 

7.1  Background 

Civil  AMVE  research  activities  in  Germany  are  as  multiple  and  manifold  as  in  other  countries.  Most  of 
these  activities  have  their  origin  in  computer  sciences  and  computer  graphics  and  are  still  positioned  here. 
But  recently,  AMVE  technology  has  become  a  tool  for  research  in  other  areas  and  applications  as  well. 
This  chapter  explained  the  general  background  of  civil  AMVE  research.  Furthermore,  the  criteria  for 
choosing  participants  in  the  survey,  as  well  as  the  send  and  return  of  questionnaires  are  described. 

7.1.1  General  Background 

Research  on  Virtual  Reality  (VR)  and  its  applications  started  around  1980.  VR  was  originally  considered 
as  an  innovative  visualization  concept  for  system  design  and  rapid  prototyping.  The  premier  goal  of  a 
natural  interaction  and  an  “intuitive”  use  made  it  a  topic  of  ergonomics  and  human-computer-interaction  at 
a  very  early  state.  Due  to  lacking  performance  it  took  several  years  until  a  reasonable  use  and  application 
became  possible.  At  first,  among  other  reasons  because  of  the  expensive  equipment  required  for  research, 
only  a  relatively  small  number  of  institutions  were  on  the  topic.  At  that  time  VR  turned  out  to  be  a  highly 
innovative  and  technology-driven  topic,  which’ s  true  application  areas  were  still  to  be  determined. 

Research  was  divided  into  then:  On  the  one  side  there  was  application-oriented  research  (focusing  on 
workplace  design,  medical  applications,  teleoperations  etc.)  and  on  the  other  side  more  basic,  fundamental 
research  on  AMVE  on  its  own  (computer  graphics,  rendering  etc.).  With  growing  availability  of  VR 
equipment,  a  broad  spectrum  of  academic,  federal,  and  industrial  institutions  started  work  in  this  publicity 
topic.  Today,  the  term  VR  has  become  widely  used  for  each  and  everything,  so  that  the  term  Virtual 
Environments  (VE)  has  become  more  popular  in  the  scientific  world. 

Due  to  higher  technological  demands,  research  on  Augmented  Reality  (AR)  and  Mixed  Reality  (MR) 
came  up  slightly  later.  Both  have  been  influenced  and  supported  largely  by  the  German  automotive 
industry.  A  special  goal  is  application  of  AR  for  development,  production  and  maintenance  of  complex 
technical  products. 

7.1.2  Criteria  for  Choosing  Participants 

A  main  problem  for  setting  up  this  questionnaire  was  to  limit  it  to  relevant  activities  in  AMVE  only. 
Traditional  simulators,  as  frequently  used  by  German  automotive  industry  and  research  were  intentionally 
left  out.  To  distinguish  between  a  traditional  simulator  on  the  one  hand,  and  a  VE-system  on  the  other  was 
difficult  because  both  overlap  widely  and  there  is  large  parallelism  between  them. 
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At  a  first  approach,  VE  is  considered  to  be  a  part  of  simulation  and  not  vice  versa.  It  is  well  known  that 
there  are  arguments  for  an  opposite  structure  as  well.  Nonetheless,  for  this  report  the  definition  of  VE 
given  in  the  introduction  and  the  notion  within  the  scope  of  the  RTG  are  strictly  referred  to.  Thus, 
simulators  were  to  be  left  out. 

Likewise  simulators,  VE  systems  visualize  synthetic,  computer-generated  databanks.  Likewise  simulators, 
VE  evoke  a  feeling  of  being  a  part  of  the  simulation.  But  unlike  simulators  VE  simulate  stimuli  to  a 
greater  amount  by  means  of  special  devices.  In  a  simulator  the  direct  surrounding  of  the  user  still  remains 
the  (real)  original  equipment  (i.e.  a  cockpit  of  a  flight  simulator  still  is  “real”  just  the  outside  view  is 
virtual)  while  in  an  ideal  VE  system  this  would  be  simulated  as  well  (i.e.  by  means  of  haptic  feedback). 
For  a  VE  system  the  synthetic  part  of  stimuli  is  considerably  larger  than  for  simulators.  Consequently, 
a  closer  user  involvement  (including  head-tracking  etc.)  and  special  hardware  is  required. 

Close  user  involvement  is  always  subjective  and  object  to  vary  largely  between  subjects.  For  this  reason  it 
seemed  unpractical  to  choose  it  as  main  criteria.  Instead,  the  specific  hardware  technology,  which  is  more 
objective,  was  chosen  to  be  the  main  criteria  for  identifying  participants  for  the  survey. 

This  -very  technocratic-  approach  gave  an  opportunity  to  distinguish  large-scale  from  small-scale 
research  activities,  because  substantial  research  in  AMVE  still  requires  special  devices  (and  expenses).  We 
have  chosen  availability  of  head-mounted-displays  (HMD),  stereoscopic  vertical  or  horizontal  projection 
surfaces  (Powerwall,  Workbench),  or  special  AMVE  hard-  and  software  as  the  participation  criteria. 
This  resulted  into  the  participating  institutions  referred  to  in  the  following  chapters. 

7.1.3  Send  and  Return  of  Questionnaires 

For  this  compendium  available  AMVE  technology,  application  areas,  and  general  areas-of-research  were 
considered  to  be  relevant.  The  questionnaire  attached  in  the  annex  was  sent  out  to  44  scientific  and 
industrial  institutions  in  Germany  and  7  in  Austria  and  Switzerland.  As  shown  in  Figure  7-1,  14  out  of 
these  were  academic/university,  12  were  federal  and  free  research  institutions,  13  industrial  laboratories 
and  5  were  developers  of  AMVE  systems. 


□  AIWE  developers 

□  Industry 

■  Independent  research 

□  Academic  /  university 


Figure  7-1:  Questionnaires  Sent  and  Received. 
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Twenty-two  questionnaires  (overall  return  rate:  50%)  were  returned.  Three  (14%  returns)  came  from 
universities,  1 1  (92%  returns)  from  federal  and  free  research,  3  (60%  returns)  from  AMVE  system 
developers  and  5  (38%  returns)  from  industrial  laboratories. 

The  results  derived  from  the  questionnaire  returns  are  summarized  in  the  following  chapters.  It  has  to  be 
mentioned  that  more  than  a  single  response  was  possible  at  every  question. 

7.2  Technology 

Technology  is  the  backbone  of  each  AMVE  activity.  Although  it  can  be  considered  as  widely  available, 
technology  development  has  not  stopped,  and  technology  trends  are  likely  to  affect  more  general  trends  in 
research.  Technology  includes  the  computing  and  graphic  platforms,  the  display  system  and  its  modality 
(visual,  acoustic,  haptic),  tracking  systems,  and  special  interaction  devices.  A  further  important  aspect  is 
software,  e.g.  operation  system,  rendering  software,  supported  databank  formats,  and  application-specific 
software. 


7.2.1  Computing  Platform 

Recently,  computing  and  graphics  power  of  personal  computers  have  tremendously  increased. 
This  resulted  in  growing  utilization  of  PCs  in  AMVE  laboratories.  They  have  widely  replaced  the 
specialized,  monolithic  graphic  workstations.  Another  effect  of  this  development  is  that  research  activities 
and  applications  of  VE  technology  have  increased  because  of  fewer  required  expenses,  though  still 
remaining  cost-intensive.  The  results  of  the  survey  are  shown  in  Figure  7-2. 


Figure  7-2:  Computing  Platform  of  AMVE-Systems. 

Single  PCs  with  high-end  graphics  cards  are  used  in  18  AMVE  institutions  (82%).  By  connecting  multiple 
PCs  into  a  cluster,  more  graphics  performance  is  available.  Fifteen  (68%)  of  the  participants  in  this  survey 
follow  this  direction.  High-end  graphic  workstations  from  Silicon  Graphics  Inc.  are  a  part  of  equipment  of 
14  institutions  (64%).  Four  (18%)  institutions  use  workstations  from  SUN  or  HP,  respectively.  For  AR  or 
MR  topics  a  wearable  PC  is  the  most  frequently  used  computer  platform  (2  cases,  9%).  The  application  of 
PC  increases  to  21  laboratories  (95%)  if  it  is  not  differentiated  between  single  or  clustered  PC. 
Consequently,  all  except  for  one  single  lab  with  a  graphics  workstation  at  least  additionally  use  PCs  in 
their  AMVE-setups. 


7.2.2  Visual  Display 

The  visual  display  plays  an  important  role  for  the  applicability  of  an  AMVE  system  because  most  of  the 
environmental  stimuli  are  perceived  by  the  optical  sense.  Figure  7-3  shows  the  visual  displays  present  at 
participating  institutions. 

As  expected,  most  participants  in  this  survey  (15  cases,  68%)  make  use  of  monitor  displays.  Thirteen 
laboratories  (59%)  are  equipped  with  HMDs  which  facilitate  a  total  immersion  into  the  VE. 
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Powerwall  (13) 

Figure  7-3:  Visual  Displays. 


The  diverse  projection-based  systems  serve  as  visual  display  in  17  cases  (77%).  Most  frequently  a  vertical 
projection  surface  is  used  (13  cases,  59%).  Other  projection-based  systems  (horizontal  projection  surfaces, 
and  multiple  surface  projection  systems)  are  each  present  in  10  laboratories  (45%).  These  systems  vary 
strongly  in  size  (from  1  to  12  m)  and  number  of  projection  planes  (1  to  6). 

In  contrast  to  this,  virtual  retinal  displays  (VRD)  (3  cases,  14%)  or  autostereoscopic  displays  (ASD) 
(2  cases,  9%)  are  only  rarely  found. 

For  projecting-based  systems  liquid  crystal  display  (LCD)  projectors  are  used  most  frequently  (6  cases, 
38%  of  projection-based  systems).  They  facilitate  a  very  bright  presentation  with  high  contrast  and  are 
relatively  low-cost.  Due  to  technical  limitations  they  require  a  time-synchronous  stereo-separation 
resulting  in  at  least  two  projectors  per  projection  surface.  Cathode-Ray  Tube  (CRT)  projectors  which  are 
used  in  5  labs  (31%  of  projection-based  systems)  work  with  higher  framerate  and  allow  time- alternating 
stereo-separation.  Nonetheless,  their  projection  is  darker  and  they  are  usually  more  expensive  than  LCD 
projectors.  Four  labs  (25%  of  projection-based  systems)  work  with  digital  light  processing  (DLP) 
projectors  that  offer  a  higher  projection  quality.  Three  labs  (21%  of  projection-based  systems)  use  Direct 
Driven  Image  Light  Amplifiers  (D-ILA)  with  higher  resolution  and  high  brightness.  Four  participants 
using  projection-based  systems  did  not  respond  to  this  question. 

Monitors  and  projection-based  displays  require  glasses  and  devices  for  stereo-separation.  These  can  be 
either  time-alternating  or  time-synchronous.  Time-alternating  shutterglasses  are  usually  found  at  monitors 
or  when  CRT-projectors  are  used.  Twelve  labs  (55%)  are  equipped  with  these  devices.  In  contrast  to 
shutterglasses  time-synchronous  anaglyph  (color)  or  polarization  filter  glasses  are  light-wear  and  cheaper. 
Polarization  filter  glasses  are  used  in  15  labs  (68%)  while  anaglyph  glasses,  which  do  not  allow  color 
presentation,  are  used  in  a  single  lab  (5%)  only. 

Other  (5)  Color  (1) 

Shutter  (12) 


Polarization  (15)  ^ 

Figure  7-4:  Devices  for  Stereo-Separation. 


7.2.3  Acoustic  Display 

Multi-modality  enhances  the  effect  of  immersion  into  the  VE.  For  this,  acoustic  displays  serve  as  an 
extension  to  visual  displays.  Acoustic  displays  are  present  in  15  institutions  (71%).  Nine  labs  (41%) 
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are  equipped  with  stereo  outputs  and  8  (36%)  use  surround  sound  systems.  Two  participants  in  the  survey 
have  both  available.  Two  institutions  use  other  acoustic  displays  (binaural,  3D  audio). 


Figure  7-5:  Acoustic  Displays. 


7.2.4  Haptic  and  Kinesthetic  Displays 

A  further  output  modality  used  by  AMVE-systems  is  haptics.  Haptics  is  especially  essential  for  fumbling, 
reaching,  and  grasping.  A  close,  natural  interaction  with  virtual  devices  becomes  difficult  without  haptic 
feedback.  Equipment  for  experimental  haptic  feedback  is  available  in  10  institutions  (45%),  as  shown  in 
Figure  7-6.  Mock-Ups  made  either  from  wood  or  other  materials  prepare  a  very  accurate  haptic  feedback 
but  are  usually  not  very  easy  to  modify.  They  are  used  in  5  labs  (23%).  Two  participants  (9%)  make  use  of 
vibrating  elements,  and  1  (5%)  use  stepper  motors  or  linkages.  Four  others  (18%)  possess  PHANTOM 
devices  that  use  stepper  motors  as  well. 


Figure  7-6:  Haptic  Displays. 

Kinesthetics  comprises  the  sensation  of  self-motion.  It  is  implemented  mostly  by  means  of  motion 
platforms,  which  simulate  accelerating  forces  for  the  operator.  Three  labs  (14%)  are  equipped  with  such 
motion  platforms. 

7.2.5  Tracking  Systems 

Receiving  and  processing  operator  inputs  is  obviously  important  for  interactive  systems.  Tracking  systems 
facilitate  tracking  of  operator’s  view  point  position  and  orientation  as  well  as  the  position  and  orientation 
of  input  devices.  Tracking  has  to  be  real-time  to  minimize  system  lags  and  dizziness.  Additionally,  they 
have  to  be  accurate  enough  to  prevent  display  errors. 

Figure  7-7  shows  that  electromagnetic  tracking  systems  are  widely  used  (16  cases,  73%),  being  followed 
by  optical  tracking  systems  (14  cases,  64%)  and  inertial  systems  (4  cases,  18%).  Two  labs  (9%)  make  use 
of  ultrasonic  devices,  and  one  possesses  equipment  for  gaze  capture.  Two  participants  gave  no  answer  to 
this  question  or  do  not  make  use  of  operator  tracking.  Again,  the  majority  of  labs  rely  on  more  than  just 
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one  method.  However,  it  is  not  clear  whether  methods  are  actively  combined  or  just  used  for  different 
setups. 


Electromagnetic 

(16) 


Inertial  (4) 


Figure  7-7:  Tracking  Systems. 


7.2.6  Interaction  Devices 

Unlike  conventional  human-computer-interaction  an  intuitive,  natural  HSI  requires  special  input  devices. 
They  vary  between  rather  simple  devices  like  mice  or  trackballs  to  highly  sophisticated  like  6-degree-of- 
freedom  (6-DOF)  controls  or  pointing  devices.  The  proportions  for  the  most  common  devices  are  shown 
in  Figure  7-8. 


Spacemouse  (17) 


Mouse  (16) 


Motion  platform  (3) 

Trackball  (5) 

Speech  (7) 

Dataglove  (10) 

Pointer  (16) 


Figure  7-8:  Interaction  Devices. 


Most  frequently,  a  6-DOF  Spacemouse  or  comparable  devices  are  used  for  3D  HSI  (17  cases,  77%). 
Sixteen  labs  (73%)  make  use  of  conventional  mice  with  just  2  degrees  of  freedom.  The  technologically 
identical  trackballs  are  found  in  5  labs  (23%). 

Pointing  devices  also  requiring  a  tracking  system  are  reported  by  16  labs  (73%).  Five  participants  (23%) 
did  not  mention  this  devise  despite  have  tracking  systems  available.  This  may  hint  at  a  misunderstanding 
of  the  question.  Therefore,  the  actual  percent  of  pointing  devices  can  be  considered  as  higher.  Data  gloves 
or  derivatives  make  gesture  input  possible  and  allow  a  very  natural  pointing  -  simply  with  the  hand 
gestures.  They  sometime  include  vibro-electric  elements  for  haptic  feedback  (see  according  section  7.2.4). 
Ten  of  the  participants’  labs  (45%)  are  equipped  with  these  gloves. 

Yet,  all  input  devices  facilitated  a  manual  operator  input  -  either  by  controlling  a  cursor,  pointing, 
or  gesture.  Speech  recognition  facilitates  acoustic  and  speech  input  and  introduces  a  further  modality. 
Speech  input  is  usually  combined  with  other  interaction  techniques  like  pointing.  Seven  institutions  (32%) 
include  speech  recognition  as  a  further  input  modality. 

In  addition  to  this,  mixed  mock-ups  (see  section  7.2.4)  frequently  use  original  input  devices  like  knobs  or 
dials  as  input  devices.  The  Phantom  device  does  not  only  serve  for  output,  but  also  for  input. 
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In  addition  to  these  commercially  available  products,  several  self-developed  devices  and  innovative 
approaches  are  used.  They  include  Cubic  Mouse,  JoJo,  Easy2C  and  others.  Special  information  about 
these  innovative  devices  can  be  found  on  the  websites  of  the  according  institutes. 

7.2.7  Operating  System 

Hardware  and  devices  described  before  are  the  backbone  of  the  AMVE  systems,  but  they  would  not 
operate  without  drivers  and  other  supporting  software.  They  are  running  on  different  operating  systems. 
Figure  7-9  shows,  that  in  14  labs  (64%)  IRIX,  the  operating  system  for  Silicon  Graphics  workstations  is 
installed.  The  same  amount  uses  LINUX  operation  system  or  Microsoft  Windows.  Only  2  labs  (9%)  run 
the  UNIX  operating  system. 


Solaris  (1) 


Windows  (14) 


IRIX  (14) 


Figure  7-9:  Operating  System. 


7.2.8  Visual  Rendering  Software 

Visual  displays  and  graphic  computing  hardware  require  a  powerful  rendering  basis.  The  proportions  of 
the  most  common  rendering  libraries  or  framework  are  illustrated  in  Figure  7-10.  OpenGL  (17  cases,  77%) 
is  used  most  often  as  basis.  Extensions  of  OpenGL,  either  self-developed  or  supplied,  are  implemented  in 
5  labs  (23%).  With  SGI  hardware  platforms  the  software  performer  is  present  in  8  cases  (36%).  DirectX 
and  derivatives  serves  as  rendering  basis  in  5  institutions  (23%). 


OpenActiveWRL 


OpenSG  (2) 

Avango  (2) 

Game-Engines  (3) 

OpenGL  ext.  (5) 


DirectX  (5) 


Other  (11)  Performer  (8) 

Figure  7-10:  Visual  Rendering  Software. 


Further  and  more  specialized  rendering  software  is  just  used  by  two  labs  (OpenSG,  Avango)  or  just  one 
lab  (Lightning,  OpenActiveWrl),  which  is  usually  the  developing  community  of  the  rendering  framework. 
Applications  of  rendering  machines  from  game-engines  were  not  reported. 
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7.2.9  Acoustic  Rendering  Software 

Likewise  visual  rendering,  there  are  special  software  frameworks  for  acoustic  rendering  as  well.  Six 
institutions  (27%)  report  activities  in  acoustic  rendering:  4  (18%)  are  using  DirectX.  2  (9%)  use  OpenAL 
as  more  sophisticated  acoustic  rendering  library. 

7.2.10  Supported  Data  Formats 

Virtual  Worlds  are  usually  not  created  online  but  stored  in  a  special  data  format.  The  rendering  software  is 
capable  to  read  this  format  and  create  a  visual  output  out  of  it. 

Answers  in  this  section  were  very  limited.  The  software  of  5  institutions  (23%)  supports  CAD  data 
formats,  and  3  (14%)  support  VRML  as  3D  modeling  language.  Other  formats  like  inventor  or  Openflight 
etc.  were  just  reported  once.  It  has  to  be  mentioned  that  the  validity  of  these  answers  is  not  high  because  of 
a  lot  of  missing  answers  (just  10  answers). 

7.3  Areas  of  Application  and  Research 

While  hard-  and  software  is  the  basis  application  and  research  gives  insights  into  the  actual  purpose  and 
intention  of  AMVE-systems.  Due  to  the  high  flexibility  of  these  systems,  both  topics  have  a  broad 
spectrum.  In  theory,  AMVE-systems  can  be  applied  to  any  possible  application.  In  reality,  there  are 
several  areas  where  the  use  of  AMVE-systems  is  very  promising  and  beneficial.  In  this  chapter  the 
relevant  software  modules  for  special  applications,  research  and  application  areas  are  described.  It  is 
shown,  that  AMVE-systems,  although  still  a  research  topics  on  their  own,  have  changed  into  an 
application  in  recent  years. 

7.3.1  Application-Dependent  Software  Modules 

Special  applications  require  more  specific  software  modules.  These  modules  can  be  either  single-stand 
modules,  or  they  are  a  part  of  complex  framework.  In  the  questionnaire  a  rough  structure  of  the  main 
application-dependent  modules  was  given.  The  result  of  the  survey  is  illustrated  in  Figure  7-11. 


Figure  7-11:  Application  Software  Modules. 

For  general  analysis  modules  6  institutions  (28%)  specified  their  software.  Main  modules  were  specific 
tools  of  Bool,  CAD,  BAF,  VD2,  ICEM/Surf,  AVANGO,  IDO  Review,  and  others.  Integrated  physical 
libraries  (5  answers,  23%)  are  ODE,  Vortex,  AVANGO,  HAVOK  1.1.  Modules  for  ergonomic  analysis  of 
workplace  design  are  available  at  7  institutions  (32%).  This  comprises  digital  human  models  like 
Anthropos  and  RAMSIS  as  well  as  specific  modules  of  VD2,  AVANGO  and  IDO-Ergonomics.  For  CAD- 
Applications  and  general  applications  in  manufacturing,  strain  analysis  and  analysis  of  flow  are  important. 
Five  institutions  (23%)  answered  this  question  positively.  Available  modules  are  within  the  AMVE- 
software  libraries  and  packages  EnSight,  AVANGO,  and  Covise. 
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7.3.2  Areas  of  Research 

Despite  of  upcoming  real  applications  AMVE  still  remains  a  research  topic  and  provides  the  infrastructure 
for  research  topics.  Only  two  institutions  (9%)  did  not  respond  to  this  question  and  are  considered  not  to 
perform  research  activities  in  connection  with  AMVE.  The  frequency  of  other  research  topics  is  shown  in 
Figure  7-12. 


Physics  (1) 

Medicine  (4) 


Figure  7-12:  Research  Topics. 


According  to  this,  the  most  frequent  area  of  research  is  human-computer-interaction  (16  cases,  73%). 
This  meets  the  expectation  because  AMVE  has  been  introduces  as  an  innovative  HSI  and  research  in  this 
area  is  required  for  optimal  usability  and  application  in  manifold  areas. 

Other  applications  make  use  of  AMVE  technology  as  a  tool.  They  are  in  the  area  of  mechanical 
engineering  /  mechatronics  (10  cases,  45%)  and  simulation  (11  cases,  50%)  of  all  kinds,  but  most 
frequently  in  combination  with  mechanical  engineering.  Nine  institutions  (41%)  use  AMVE  technology 
for  the  visualization  of  massive  datasets.  In  4  labs  (18%)  medical  applications  of  AMVE  technology  are 
the  main  focus.  Applications  in  combination  with  other  sciences  like  physics  (1  case,  5%)  or  chemistry 
(no  case)  can  be  neglected.  Further  application  areas  (1  case  per  area)  are  architecture  and  modeling, 
storytelling,  basic  research  in  perception,  and  team  training. 

7.3.3  General  AMVE  Technological  Development 

In  addition  to  research  in  more  application-oriented  areas,  AMVE  itself  is  being  developed  further  on. 
This  is  necessary  because  of  changing  requirements  for  experimental  setups  and  applications  as  well  as 
changing  technical  capabilities.  In  total  16  institutions  (73%)  report  activities  in  general  technological 
developments.  As  shown  in  Figure  7-13,  software  development  is  the  main  goal  of  development.  Six  labs 
(27%)  develop  both,  new  software  and  hardware  for  AMVE.  Two  labs  (9%)  develop  new  hardware  and 
9  (41%)  new  software  and  software  modules  only. 


Figure  7-13:  AMVE-Technology  Development. 
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7.3.4  Product  Design  and  Development 

In  product  development  AMVE  technology  is  used  primarily  as  a  tool.  In  this  case,  AMVE  provides 
infrastructure  for  developing  a  diversity  of  other  systems.  Figure  7-14  shows  that  most  frequently 
industrial  manufacturing  processes,  car  manufacturing,  and  simulation  (10  cases,  45%)  are  reported  as 
application  area.  The  following  item,  including  7  cases  (32%),  is  product  presentation.  Five  institutions 
(23%)  use  AMVE  technology  for  application  in  architecture,  aviation,  and  product  design.  Other 
applications  areas  are  education  and  training  in  maintenance  (1),  automatisation  (1),  and  military 
applications  (1). 


Simulation  (10) 


Other  (3)  Architecture  (5) 

Aviation  (5) 


Automobile  (1 0) 


Industry  (10) 

Figure  7-14:  Area  of  Product  Design  and  Development 


Design  (5) 
Presentation  (7) 


8.0  CONCLUSION 

The  report  describes  applications  and  research  in  institutions  and  within  larger  projects  in  Germany. 
Furthermore,  a  brief  sketch  of  technological  and  thematic  characteristics  and  trends  for  the  civil  sector  is 
given.  It  shows  that  AMVE  systems  have  evolved  from  a  pure  research  topic  to  an  application  topic  and 
that  this  trend  is  about  to  grow  in  future. 

One  obvious  finding  is  the  broad  usage  of  PC  systems  for  AMVE  activities.  Workstations  are  still  used  but 
even  then  PCs  are  used  as  an  addition.  The  use  of  COTs  hardware  can  be  seen  in  display  technology  as 
well.  Monitors  are  most  often  used,  being  followed  by  projection-based  systems.  Of  course,  the  latter 
cannot  be  considered  as  commercially  widely  available,  but  they  were  strongly  affected  by  dropping  prices 
for  LCD  projection  systems.  Moreover,  they  do  not  have  handling  or  comfort  problems  like  HMDs. 
And  they  make  team  activities  possible;  something  that  is  obviously  impossible  with  a  single  HMD  only. 

As  expected,  visual  presentation  is  more  frequently  present  than  acoustic  or  other  modalities.  However, 
acoustic  is  growing  and  available  at  several  setups. 

Tracking  is  widely  done  by  optical  or  electromagnetic  systems.  Interaction  devices  are  manifold  and 
include  simple  as  well  as  highly  complex  devices  of  various  kinds. 

With  regard  to  software,  Window,  Linux  and  IRIX  are  most  common.  For  acoustic  or  visual  rendering 
most  frequently  self-developed  software-frameworks  or  open-source  software  is  used.  They  usually  have 
benefits  for  specific  applications  but  are  variable  enough  to  adjust  them  to  others. 

The  main  application  area  is  ergonomics  and  within  this,  human-computer-interaction.  Other  areas  are 
simulation,  mechatronics,  and  visualization.  Consequently,  modules  for  physics,  analysis  of  various  kinds, 
and  flow  are  available.  Considering  AMVE  systems  on  their  own,  software  is  more  often  developed  than 
hardware. 

Most  common  application  field  is  simulation,  automotive  and  general  industry,  and  product  presentation. 
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In  addition  to  the  national  findings,  the  NATO  RTG  has  identified  emerging  technologies  in  one  of  its 
recent  meetings.  All  of  them  will  have  effects  on  AMVE  technology  and  its  application.  This  includes: 

•  Vision  (laser-based  projection  systems,  small  dome -projection  systems,  holographic  displays, 
autostereoscopic  displays) 

•  3D-audio  (new  real-time  software) 

•  Motion  platforms  (new  pneumatic  and  electromechanic  systems) 

•  Vibro-tactile  elements  (haptic  feedback) 

Seen  from  an  international  view,  one  of  the  main  applications  of  AMVE  technology  still  remains 
simulation  and  training.  This  is  not  observed  in  Germany,  but  is  likely  to  develop  with  growing 
availability  of  AMVE-systems.  This  will  result  into  more  detailed  modeling  of  the  synthetic  environment 
and  the  development  of  more  realistic  simulation  environments.  It  also  includes  intelligent  agents, 
i.e.  behavior  models,  CGF,  virtual  combatants  and  their  dynamic  behaviors  becoming  a  part  of  the  training 
environment  as  well. 

A  further  general  trend  is  based  on  the  falling  prices  of  AMVE  hardware.  Increased  computer  power  at 
low  cost,  wireless  networks,  shrinking  of  computer  components,  and  better  visual  and  auditory  display 
systems  are  contributing  to  the  maturation  of  these  technologies.  Potential  applications  in  military 
operations,  as  well  as  training  and  system  design  are  providing  requirements  that  have  spurred  the 
technologies’  development.  Still  most  of  the  attention  is  drawn  on  the  development  of  the  technologies 
themselves.  However,  to  be  effective  in  military  operations,  the  technologies  must  evolve  into  reliable 
systems  that  provide  the  information  that  their  human  users  need  to  accomplish  task  objectives. 

Compared  to  research  on  computer  architectures,  communication  protocols,  and  display  devices,  there  has 
been  relatively  little  research  on  the  perceptual  requirements  for  displays,  HSI  issues  from  a  human- factors 
point-of-view,  design  of  effective  training  approaches,  and  measurement  of  human  performance. 
The  fundamental  knowledge  available  today  already  indicates  a  large  potential  of  AMVE  technology  for  a 
broad  spectrum  of  military  applications,  including:  1)  dynamic,  task-driven  user  interfaces  for  C4ISR 
systems;  2)  telepresence,  teleoperation,  and  telemanipulation  in  reconnaissance,  surveillance,  and  target 
acquisition;  3)  realistic,  distributed  military  simulation  and  training;  4)  mission  preparation  en-route, 
including  intended  area  of  operation;  and  5)  mission  support  as  wearable,  augmenting  technology  for 
individual  operators. 
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Appendix  A:  Participating  Institutions 

We  would  like  to  thank  the  following  institutions  and  persons  for  their  participation  and  help  in  the 
survey. 

Bauhaus  Universitat  Weimar 
Bauhausstr.  11 
99423  Weimar 
Prof.  Dr.  Bemd  Frohlich 

Deutsches  Zentrum  fur  Luft-  und  Raumfahrt 
Lilienthalplatz  7,  PF  3267 
38022  Braunschweig 
Dr.  phil.  nat.  Norbert  Fiirstenau 

Dr.  Ing.  h.c.  F.  Porsche  AG 
Porschestr. 

71287  Weissach 
Ingo  Krems 

EADS  Deutschland  GmbH 
8 1 663  Miinchen 
Leonard  Vogelmeier 

FGAN-FKIE 
Neuenahrer  Str.  20 
53343  Wachtberg 
Dr.  Thomas  Alexander 

Fraunhofer  Institut  fur  Arbeitswirtschaft  und  Organisation  (IAO) 

Nobelstr.  12 

70569  Stuttgart 

Dr.  Manfred  Dangelmaier 

Fraunhofer  Institut  fur  Fabrikbetrieb  und  -automatisierung  (IFF) 

Sandtorstr.  22 

39106  Magdeburg 

Dr.  rer.  nat.  Eberhard  Bliimel 

Fraunhofer  Institut  fur  Medienkommunikation 
SchloB  Birlinghofen 
53754  Sankt  Augustin 
Dr.  Martin  Gobel 

Fraunhofer  Institut  fur  Rechenarchitektur  und  Softwaretechnik  (FIRST) 

Kekulestr.  7 
12489  Berlin 
Herbert  Riisseler 

Fraunhofer  IWU  /  TU  Chemnitz 
Reichenhainer  Str.  88 
09126  Chemnitz 
Prof.  Dr. -Ing.  Dieter  Weidlich 
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Heinz  Nixdorf  Institut,  Universitat  Paderbom 

Fiirstenallee  1 1 

33102  Paderbom 

Prof.  Dr.  Jurgen  Gausemeier 

Institut  fur  Betriebs-  und  Dialogsysteme,  Universitat  Karlsruhe 
Am  Fasanengarten  5 
76128  Karlsruhe 
Sven  Thuring 

MPI  fur  biologische  Kybemetik 

Spemannstr.  38 

72076  Tubingen 

Prof.  Dr.  Heinrich  H.  Biilthoff 

Ray  Sono  AG 
Otto-Lilienthal-Str.  8 
28199  Bremen 
Klaus  Bock-Muller 

Realtime  Technology  AG 
Rosenheimer  Str.  145f 
81671  Miinchen 

Research  Center  CAESAR,  Surgical  Systems  Lab 
Ludwig-Erhard  Allee  2 
53175  Bonn 
Dr.  Erwin  Keeve 

RTWH  Aachen 

Institut  fur  Arbeitswissenschaft 
Bergdriesch  27 
52062  Aachen 
Ludger  Schmidt 

RWTH  Aachen 
Rechenzentrum 
Seffenter  Weg  23 
52074  Aachen 
Dr.  Torsten  Kuhlen 

Siemens  AG 
PF  4848 
90327  Niirnberg 
Wolfgang  Friedrich 

Viception  GmbH  &  Co.  KG 
Schleifacker  88 
89081  Ulm 
Dr.  Walter  Gillner 

Volkswagen  AG 
PF  1511,  VR-Labor 
38436  Wolfsburg 
Peter  Zimmermann 
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A  special  thanks  to 

Leonard  Vogelmeier 
EADS  Deutschland  GmbH 
8 1 663  Miinchen 

for  providing  us  with  chapter  3 

and 

Friedhelm  Reiners 

IABG  Dept.  “Networked  Simulations” 

Schiessplatz 

49716  Meppen 

for  providing  us  with  chapter  4. 


Information  considering  chapter  5  ( Co-operative  Computer-Based  Training  for  the  Amphibian  M3) 
is  based  on  the  internet  domain  of  the  project  -  http://www.kocua.info/. 

•  Contact  for  further  information:  Ray  Sono  AG  Bremen,  Otto-Lilienthal-Str.  8,  D-28199  Bremen 

Information  considering  chapter  6  {Advanced  Air  Defence  Training  Simulation  System)  is  based  on  the 
www-homepage  of  EADS  -  http://www.eads.com/ 

•  Contact  for  further  information:  EADS  Defence  and  Security  Systems,  Business  Unit  Systems  & 
Defence  Electronics,  88039  Friedrichshafen 
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